Imitation in Reinforcement Learning

The promise of imitation is to facilitate learning by allowing the learner to observe a teacher in action. Ideally this will lead to faster learning when the expert knows an optimal policy. Imitating a suboptimal teacher may slow learning, but it should not prevent the student from surpassing the teacher’s performance in the long run. Several researchers have looked at imitation in the context of reinforcement learning. Perhaps the most straightforward formulation is to apply a standard reinforcement learning algorithm such as Q-learning to the teacher’s experience rather than the learner’s [9, 2, 3, 8]. Price and Boutilier extend this approach to the case where the learner does not know what actions the teacher takes [6, 7]. They do assume, however, that the learner knows a priori its reward R(s) for a transition into any state s. Another way to incorporate expert information is shaping. Shaping is the introduction of small rewards into certain states where progress towards an environment reward is made. Shaping can speed up learning, but runs the risk of corrupting the underlying reward structure and thus changing which strategies are optimal. Ng et al. have shown that potential-based shaping functions preserve the partial ordering of policies with respect to optimality [4]. This means poor potential functions will at worst slow learning, but will not prevent convergence in the long run; potential functions which closely approximate states’ true values will speed up learning. An ideal potential function for imitation would be based on salient properties of the teacher’s policy. If the teacher consistently completes some set of subgoals in the process of receiving a reward, it would be beneficial to receive shaping rewards when the learner completes these. Ng is in the process of developing a way to extract an learner’s rewards from observations of its policy, but his method does not guarantee a reward function that makes a good shaping function: it favors larger sporadic rewards over smaller, frequent rewards [5]. In fact, Ng poses reverse engineering a shaping function as an open problem.